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Abstract. We provide geometric conditions on the set of boundary points of in- 
finite type of a smooth bounded pseudoconvex domain in C™ which imply that 
the 9-Neumann operator is compact. These conditions are formulated in terms of 
certain short time flows in suitable complex tangential directions. It is notewor- 
thy that compactness is not established via the known potential theoretic sufficient 
conditions. Our results generalize to C™ the C 2 results from 



1. Introduction 

In (3T| , the second author provided geometric sufficient conditions for compactness 
of the <9-Neumann operator on the boundary points of infinite type of a bounded 
smooth pseudoconvex domain in C 2 . In this paper, we study the situation in higher 
dimensions and obtain suitable generalizations of the results in j!H] . 

Let Q be a bounded pseudoconvex domain in C n . The d- Neumann operator N q 
on (0, g)-forms is the inverse of the complex Laplacian dd + d d associated to the 
Dolbeault complex. For detailed information on the d- Neumann problem and related 
questions, see e.g. |H1 IH HH1 12 123 E2]; compactness of N q is discussed in [TT1 152] . 

Whether or not N q is compact is relevant in a number of situations. These in- 
clude global regularity [T7] , the Fredholm theory of Toeplitz operators [T3] , and the 
(non) existence of solution operators to d with well-behaved solution kernels [Hj. 
There are also interesting connections to the theory of Schrodinger operators ^21 IH] • 
Catlin gave a sufficient condition, which he called property (P), in [3]: there should 
exist, near the boundary, plurisubharmonic functions bounded between and 1, with 
arbitrarily large Hessians. (The smoothness assumptions on the boundary of the 
domain were considerably weakened in [T^], and were completely removed in [30.) 
Property(P) was studied in detail (under the name P-regularity) by Sibony in [2E] 
(see also j25])- On sufficiently regular domains, property(P) is equivalent to a quan- 
titative version of Oka's lemma ( A version of property (P), called condition 
(P), was introduced, and shown to still imply compactness, by McNeal in [22] • The 
uniform bound on the family of functions is replaced by a uniform bound on the 
gradient, measured in the metric induced by the complex Hessian of the functions. 
(Both (P) and (P) can also be formulated naturally at the level of (0, g)-forms; then 
(P q ) => (P q+1 ), (P q ) => (P 9+ i), and (P q ) (P q ), 1 < g < n, see HUI221E2j.) A 
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sufficient condition that is intermediate, in a sense one can make precise (see the 
discussion in [22] )> had appeared earlier in |35j . 

On locally convexifiable domains, (P q ) and (P q ) are equivalent, and equivalent to 
compactness of N q , for 1 < q < n. Moreover, the three properties are equivalent 
to a simple geometric condition, the absence of (germs of) g-dimensional varieties 
from the boundary. For this, see JUJ E] • Thus the potential theory, the analysis, 
and the geometry mesh perfectly on locally convexifiable domains. It is also known 
that on smooth bounded Hartogs domains in C 2 , compactness of jVj is equivalent 
to (Pi) and to (Pi) (0 E2])- However, it is well understood that the boundaries 
of convex (hence of locally convexifiable) domains do not exhibit some of the more 
intriguing aspects of the interaction with the ambient space that occur on general 
pseudoconvex boundaries (;6j). For example, matters concerning orders of contact 
are always decided by orders of contact of manifolds (affine manifolds in the convex 
case) ( j2H IH HUH EH ) . A similar caveat applies in the case of domains in C 2 ( 6 ) . 
As a result, these facts give no clear indication of how much (or how little) room 
there is, in the general pseudoconvex case, between (P)/(P) and compactness of the 
<9-Neumann operator. (As far as we know, the exact relationship between (P) and 
(P) is also unknown, but we do not address this question here.) 

From the point of view of obstructions, the situation is as follows, g-dimensional 
varieties in the boundary are an obstruction for both property(P g ) and condition (P^) 

(EHUD El for (P,), [H2II11 for (P q )). With respect to compactness of N q , less is 
known. A g-dimensional complex manifold M in the boundary of a smooth bounded 
pseudoconvex domain is known to be an obstruction to compactness of N q , provided 
M contains a point at which the domain is strictly pseudoconvex in the directions 
transverse to M ( J2tjU27| ). It is open whether the conclusion holds without assuming 
that there is such a point. One would expect that a flatter boundary is even more 
favorable for noncompactness, but the methods of [2Z] do not seem strong enough 
to yield noncompactness without some additional assumption on how M sits inside 
the boundary. On the other hand, both for (P) / (P) and for compactness of the d- 
Neumann operator, there are obstructions more subtle than varieties in the boundary 
( [2E1 1201 E] ) • For a connection of some of these issues with properties of the Kobayashi 
metric, see [TT)] . 

pH] provided, for the first time, a method to prove compactness of the <9-Neumann 
operator that does not proceed via verifying property(P) or condition (P). That the 
dimension is two was only used in the application of so-called maximal estimates. 
Consequently, the results of [2] hold more generally on domains in C n , n > 2, where 
such estimates hold, or, equivalently, on domains where all the eigenvalues of the 
Levi form are comparable IB]. However, for the problem of compactness of the 
<9-Neumann operator, this assumption is too restrictive. It excludes, for example, the 
situation where the Levi form has at most one degenerate eigenvalue (see Remark 
5 below). But this assumption on the Levi form has been shown to be a useful 
generalization of the case of C 2 in the context of compactness (|26[ I27j). 

The obvious examples that satisfy the assumptions in Theorem ^below also satisfy 
property(P); we do not know whether the theorem can actually furnish examples 
of domains where the <9-Neumann operator is compact, but where (P) fails. But 
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just as in [21], we obtain a simple geometric proof of compactness in these cases. 
Moreover,the assumptions are in some instances 'minimal': they are necessary modulo 
the size of certain balls; see Remark 6 below for details. 

We will only consider the case q = 1 in the remainder of this paper. This is the 
main case in terms of understanding compactness. But note that compactness of N\ 
implies compactness of N q for q > 1; this is an observation due to McNeal (|2~lj. see 
also the proof of Lemma 2 in [22] for a related argument). 

The bulk of this paper represents a portion of the first author's Ph.D. thesis ([25 ) 
written at Texas A&M University under the direction of the second author. 

2. Results 

If Z is a (real) vector field defined in some open subset of the boundary (or of C n ), 
we denote by T l z the flow generated by Z . We use the notion of finite or infinite type 
of D'Angelo [E]. For a boundary point (, we denote by Ao(C) the smallest eigenvalue of 
the Levi form of the boundary at (. Since the domains in question are pseudoconvex, 
^o(C) > 0- F° r a se t °f re& l vector fields T±, • • • ,T m on an open subset of bQ (or of 
C n ), we define span^iTx, • • , T m ) to be the set of all linear combinations of Tx, ■ ■ ■ ,T m 
whose coefficients are real valued functions (not necessarily constants). 

Theorem 1. Let Q be a C°° -smooth bounded pseudoconvex domain in C™. Denote by 
K the set of boundary points of infinite type. Assume that there exist smooth complex 
tangential vector fields Xx, ■ ■ ■ X m , defined on bfl near K, so that H p (Xi((), Xi(Q) 
< CAo(C); for some constant C , a sequence {tj}°° =1 with limtj = 0, and constants 

Cij C*2 > 0, C3 with 1 < C3 < so that the following holds. For every j £ N 
and p £ K there is a real vector field Z p j £ span R (ReXi, ImXx, ■ ■ ■ , ReX m , ImX m ) 
of unit length, defined in some neighborhood of p in bfl with max\divZ p j\ < Cx, such 
that J r z p (B(p,C2(ej) C3 ) D K^j C 60 \ K. Then the d-Neumann operator Nx on Q is 
compact. 

Remark 1: In C 2 , take m = 1 and Xx = L, where L is a smooth nonvanishing 
complex tangential vector field of of type (1, 0) on bfL Then the condition on the 
vector fields Z p j becomes simply that they be complex tangential, as in [21] . Theorem 

therefore generalizes the main result in [21]. 

Remark 2: In comparison to the main result in [21], Theorem ^ contains the 
additional assumption that there is a family of vector fields Xx ■ ■ ■ X m satisfying 

(1) H p (Xj((),X-((j)<C\ (0, 

such that the vector fields Z p j are contained in the linear span (over R, in the sense 
of the definition above) of the real and imaginary parts of these fields (as opposed 
to just being complex tangential). Without some assumption on the fields Z p j more 
restrictive than being complex tangential, such as the one made here, the result does 
not generalize to C n . To see this, consider a smooth bounded convex domain in C 3 
which is strictly convex, except for an analytic (affine) disc in the boundary. Then 
one can flow along complex tangential directions from points of the disc into the set 
of strictly (pseudo) convex boundary points as required in the second part of the 
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assumption in the theorem. Nonetheless, because there is a disc in the boundary, the 
<9-Neumann operator on (0, l)-forms is not compact on such a domain (see [TU]). 

Remark 3: In the example in Remark 2, the value of the Levi form in the direction 
parallel to the disc apparently goes to zero faster, upon approach to the disc, than 
the value in the direction transverse to the disc. Thus (JIJ cannot hold for a vector 
field transverse to the disc, and Theorem^ does not apply. It would be very desirable 
to have a direct proof of this in general, i.e. a proof that the assumptions in Theorem 
[0 exclude discs from the boundary (or, less likely, a counterexample). This is relevant 
because the question whether in general, discs in the boundary are obstructions to 
compactness is open (compare the discussion in the introduction for what is known). 
Note that in C 2 , such discs are known to be obstructions to compactness ([11 ); in 
this case, it is also obvious that the assumptions in the theorem exclude discs from 
the boundary. 

Remark 4 '■ Because compactness of the d- Neumann operator follows from a com- 
pactness estimate for forms supported in fixed, but possibly small, neighborhoods of 
boundary points (see e.g. JI]), there is a version of Theorem^where the assumptions 
are localized. For this, see the discussion in Example 3 below. 

In Theorem^ one would like to have a collection of vector fields {Xj}™ =1 such that 
at each point p of K, spariR{ReXi(p) , ImXi(p), • • • , ReX m (p), ImX m (p)} is as 'big 
as possible', thus putting the least restrictions on the fields Z p j. On the other hand, 
this needs to be balanced with the requirement {1}. We now discuss some examples. 

Example 1 : When the eigenvalues of the Levi form are all comparable, any finite 
collection of complex tangential vector fields X\, . . . ,X m will satisfy condition ([TJ. 
Taking a collection which at each point p G K spans all of T^(p), we see that in this 
case, this part of the assumptions in Theorem ^ reduces to Z p j complex tangential 
(as in C 2 ). Domains where all the eigenvalues of the Levi form are comparable play 
a special role in the theory of the 9-Neumann problem; certain estimates, called 
'maximal estimates', hold. This class of domains was studied in detail in [7||H]. 

Example 2: Assume there exists a smooth complex tangential vector field X\ near 
K such that for £ G K, X\(() is an eigenvector associated with the smallest eigenvalue 
of the Levi form at £. This vector field trivially satisfies the condition Then the 
assumption in Theorem ^ requires that Z p j(() is in the two real- dimensional plane 
spanned by Xi(Q for all (. 

Example 3: Assume that the Levi form has at most one degenerate eigenvalue at 
each point of K (hence near K). Fix a point p G K. Choose an (n — 2)-dimensional 
subspace of T^(p) on which the Levi form is strictly positive, and choose a basis. 
Extending the basis vectors to local sections of gives vector fields (defined near p) 
Y2, ■ ■ ■ , Y n -\- In a neighborhood of p, the Levi form is strictly positive on the span of 
Y 2 , - ■ ■ , Y n -\- As a consequence, at each point there is a unique one dimensional sub- 
space of T 6 fj that is orthogonal to this span with respect to the Levi form. Indeed, if Y\ 
is such that Yi, • • • , F n _i is a basis for (near p), then X 1 = Yi + b 2 Y 2 ■ ■ •-\-b n -{Y n _ 1 , 
where bj = —H p (Y 1 , Yj)/H p (Yj, Yj), j = 2, • • • , n — 1, will span this subspace. (This 
was observed already in Lemma 2.1.) Note that when p G K, then the eigen- 
vector of the Levi form at p associated with the eigenvalue zero is orthogonal to all 
of T^(p) with respect to the Levi form (because the Levi form is positive definite). 
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Therefore,by uniqueness, X\(p) is an eigenvector of the Levi form at p with eigenvalue 
zero. As a result, H p (Xi((), Xi(()) < CAo(C) for ( close enough top (by continuity), 
that is, holds for the family consisting of the single field X\. We note that when ( 
is a strictly pseudoconvex point, Xi(Q need not be an eigenvector of the Levi form. 
By multiplying with a cutoff function that is identically equal to one near p, we may 
assume that Xi(() is defined on all of bQ, with (0) still valid. Of course, this is at 
the expense of having trivial span on a big set. We may proceed in two ways. We 
can cover K with finitely many open sets Ui, ■ ■ ■ ,U S on which the cutoff functions 
multiplying these local fields are one, and then add finitely many of the fields. How- 
ever, the resulting field may still vanish at some points of K. Alternatively, we can 
take advantage of the fact that compactness localizes: it suffices to prove a compact- 
ness estimate for forms supported in (small) neighborhoods of boundary points (as 
long as the neighborhood do not depend on the e in the compactness estimate). The 
proof of Theorem ^ using the field Xj , gives a compactness estimate for forms whose 
support meets the boundary in one of the U/s. Since the U/s cover K, the result is 
a compactness estimate for all forms in dom{d) D dom(d ) (since away from K, we 
have subelliptic estimates). 

Remark 5: On the domains discussed in Example 3, the Levi form is locally di- 
agonalizable (use Gram-Schmitt to or t honor malize Y 2 , ■ • • , Y n -\ with respect to the 
Levi form). Derridj showed in (jHj, Theorem 7.1), that if maximal estimates hold at 
p G bQ, and p is a weakly pseudoconvex point, then the Levi form of Q cannot be 
diagonalizable near p when Q is a domain in C n with n > 3. Therefore, the examples 
(1) and (3) are mutually exclusive (when n > 3). 

Remark 6: It is interesting to note that in Example 3, the assumptions in the 
theorem are 'minimal', that is, they are necessary modulo the size of the balls 
B(p,C2(ej) C3 ). The discussion is analogous to the one in Remarks 2 and 3 in [31, 
but using recent results from For p G K, let X\ be the complex tangential field 
from Example 3 above (defined near p, with H p (Xi(p), Xi(p)) = 0). Denote by T e 
the field T e = cos(9)ReX 1 + sin(9)ImX 1 . For ( near p, set M ce = {^(C)|0 < 
9 < 27r, < t < t }. Then is a smooth two dimensional submanifold of the 
boundary. Because Ni is compact, the boundary contains no analytic discs (since the 
Levi form has at most one degenerate eigenvalue, see Theorem 1). Therefore, 
Lemma 3 in [2Zj implies that there exist points ( G M T e arbitrarily close to p with 
H p (Xi((), Xi(()) > 0. Because of the way X\ was constructed, such a point ( is 
a strictly pseudoconvex point. Consequently, for €j > and ( near p, there exist 
real fields Z^j G span^(ReX 1 , ImXi) of unit length so that J 7 z (j ( z ) ^ K ioi z close 
enough to (. This yields balls as in the theorem, but without control of the radii from 
below in terms of 6j. Because of the form of Z^j, the uniform boundedness condition 
on the divergence of the fields Z^- is satisfied (near p). 

We say that bfl \ K satisfies a complex tangential cone condition if there exists a 
finite open real cone T in C n ~ M 2n such that the following holds. For each p G K 
there exists a complex tangential direction so that when V is moved by a rigid motion 
to have vertex at p and axis in that direction (that is, in the two dimensional real 
affine subspace determined by that direction), the (open) cone obtained intersects bfl 
in a set contained in bfl \ K. Theorem has the following corollaries. 
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Corollary 1. Let Q be a C°° -smooth bounded pseudoconvex domain in C n . Denote 
by K the set of boundary points of infinite type. For all points ( in a neighborhood of 
K in bQ, denote by Ao(C) the smallest eigenvalue of the Levi form. Assume that bQ 
satisfies the following conditions. There exist smooth complex tangential vector fields 
Xi, . . . , X m , defined on bQ near K so that H p (Xi(Q, Xi(Q) < C Ao(C), for some con- 
stant C and all (, such that bQ\K satisfies a complex tangential cone condition with 
the axis of the cone at p G K in span R (ReXi(p), ImXi(p), . . . , ReX m (p), ImX m {p)), 
for all p G K . Then the d -Neumann operator on Q is compact. 

In C 2 , the assumption in Corollary ^ reduces to the simple requirement that K 
satisfy a complex tangential cone condition; this is Corollary 2 in (31]. An example 
like the one described after the statement of Theorem^shows that this is not sufficient 
in C" when n > 3, not even when one assumes that the axis of the cone at p G K 
lies in the null space of the Levi form at p. So some complication in the statement 
of the corollary cannot be avoided. On the other hand, when the Levi form of bQ 
has at most one degenerate eigenvalue, there is (locally) a complex tangential vector 
field X\ and a constant such that H p (Xi(p), Xi(p)) = and H p (X\,X\) < CXq near 
p G K (see Example 3 above). With this additional information, it suffices to require 
that the axis of the cone lie in the null space of the Levi form. 

Corollary 2. Let Q be a smooth bounded pseudoconvex domain in C n ; assume that 
at each boundary point, the Levi form has at most one degenerate eigenvalue. If the 
set bQ\ K satisfies a complex tangential cone condition with the axis of the cone at 
p G K in the null space of the Levi form at p, then the d-Neumann operator on Q is 
compact. 

In C 2 , Corollary El also reduces to Corollary 2 in [3T] . 

3. Proof of Theorem [T] 

We will establish a compactness estimate for forms in C^ X JQ) n dom(d*)\ for all 

e > 0, there is a constant C e such that ||w|| 2 < e(||<9w|| 2 + ||<9*w|| 2 ) + Ce||w|| 2 1 . This is 
equivalent to compactness of Ni (see for example [H], Lemma 1.1). 

First note that we may assume that the vector fields X/~, 1 < k < m, are defined 
on all of bQ, by multiplying them with suitable cutoff functions that are identically 
equal to one near K. This preserves (0) (i.e. now holds on all of bQ). Then 
we can extend the vector fields Z p j and X^ from bQ to the inside of Q by letting 
them be constant along the real normal, so that the extended fields, still denoted by 
Z p j and X k , respectively, are complex tangential to the level sets of the boundary 
distance. Finally, by multiplying by a suitable cutoff function that equals one near 
the boundary, we may assume that the fields X k are defined and smooth on Q. 

There are two ideas in the proof. The first comes from (21] and says that near a 
point p G K, the values of a form u can be expressed by the values near . (p) plus 

the integrals of Z p ju along the integral curves of Z p j. If points near J 7 J ,{p) are of 
finite type, the contribution from there can easily be estimated in the required manner 
by subelliptic estimates. Because the integrals of Z p ju in the second contribution 
are over curves of length 6j (\Z p j\ = 1), a (small) factor ej appears (via the Cauchy- 
Schwarz inequality) when computing the relevant £ 2 -norms. There are overlap issues, 
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but these are handled by the uniformity built into the assumptions (for example 
the uniform bound on divZ p j). The second idea concerns control of ||Z p ju|| 2 , or 
ultimately ||X fc -u|| 2 , by ||<9w|| 2 + ||<9*-u|| 2 . In C 2 , this can be done for any complex 
tangential field, via maximal estimates (see In higher dimensions, additional 

assumptions are needed; that (JTJ) suffices is implicit in work of Derridj ([Zj)- 

The first part of the proof follows [31] verbatim, with only one obvious modification. 
Fix e > and choose j big enough so that €j < e. The arguments in [3T], pp. 705-708 
give estimate (J2J) below, the only modification being the exponent of €j, which depends 
on the dimension. This dimension dependence arises from a comparison of volumes 
argument used to resolve certain overlap issues; see the paragraph in |3T| that starts 
at the bottom of page 707. Note that when n = 2, In + 2 — 2nC^ = 6 — 4C*3, as in 
[3T] . equation (14). Combining (the analogues of) equations (6) and (14) in |3l] gives 



\u\ 2 < 2e(||<9u|| 2 + ||«9* M || 2 ) + C e \\u\W + 4e, / C(C 2 )(e 



) 2n ~ 2n0i 2mx 



(2) 



£ 

k=l 



\ReX k u(y)\ 2 + \1mX k u(y)\' 



dV{y) dt 



< 2e(\\du\\ 2 + \\d*u\\ 2 ) + C e \\u\W + 16mC(C 2 )(e, 



\2n+2-2nC 3 



X 



E / (\X k u{y)\ 2 +\X k u(y)AdV(y)\dt . 



In C 2 , the last term on the right hand side of (0) can be estimated using maximal 
estimates ([7]): 



(3) \\X k uf + \\X k uf < C k [\\du\\l + \\d*u\\l) . 

We are going to show that (0) also holds under hypothesis on the vector fields 
X k . The estimates below owe much to jjj. For notational convenience, fix k and 
denote X k by X. First note that the estimate on ||Xu|| 2 follows directly from the 
Kohn-Morrey formula ([3], Proposition 4.3.1 with weight e - * = 1). For ||X-u|| 2 , we 
integrate by parts to obtain 

\\Xu\\ 2 = - I uXX^ + 0(\\u\\ \\Xu\\) 
Jq 

(4) = - j u[X,X]u- j uXXu + 0{\\u\\\\Xu\\) 

Jn Jq 

= - [ u [X,X~]u+\\Xu\\ 2 + 0(\\u\\ \\Xu\\ + \\u\\ \\Xu\\) . 
Jn 

Using that \\u\\ 2 < (\\du\\ 2 + \\d*u\\ 2 ), \\\u\\ \\Xu\\ < l.c\\u\\ 2 + s.c\\Xu\\ 2 (0, (4.4.6) 
on page 79), and again the Kohn-Morrey formula for ||Xm|| 2 , we see that it suffices 
to estimate the first term on the last line of Here, s.c and I.e. denote small and 
large constants, respectively; the term ||Xm|| 2 on the right hand side in Q can then 
be absorbed into the left hand side. 
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Near bfl (we may assume on the support of the cutoff functions used at the begin- 
ning of this section to extend X to all of 0), we write [X, X~\ u as H P (X, X)(L n — 
L n )u + Au + Bu, where A and B are smooth complex tangential fields and L n is the 
complex normal Y^=i(®P/ '9^)6 '/ 'dzj appropriately normalized. The contributions 
coming from Au and L n u, respectively, (or Au and L n u) are estimated as above. In 
the contribution from Bu, we integrate B by parts, and proceed as before. We are left 
with estimating the term involving H p (X, X)L n u. We integrate L n by parts (with a 
boundary term) to obtain 



(5) 



/ uH p {X,X)L n u < \\L n u\\\\u\\ + [ H p (X,X)\u\ 2 + 0(\\u\ 
Jn Jbn 



The first and the third term on the right hand side of (jSJ) are estimated as above, 
while for the middle term we have 

(6) / H p (X,X)\u\ 2 < [ X \u\ 2 <[ H p (u,u)<\\du\\ 2 +\\d\\\ 2 . 
Jbn Jbn Jbn 

The first inequality in (jUJ) comes from (Q), the last one from the Kohn-Morrey formula. 
We have slightly abused notation in the third term in (jHJ): u is a (0, l)-form, not a 
vector field, but it is identified with a vector field in the usual way via its coefficients. 
Combining (jlj) , © , and © shows that (jSJ) holds under the assumptions of Theorem 
[U (i.e. under ((H)). Inserting Q into © gives 

(7) I \u\ 2 < C(e + e 2n+2 - 2nC *) (\\8u\\ 2 + \\B\f) + C e \\u\\\ , 

with C independent of e. Since 2n + 2 — 2nCz > (because C3 < (n + l)/n ), 
lim e ^ + C(e + e 2n+2-2nC 3 ^ = 0, so ((TJ) implies the compactness estimate we set out to 
prove. This completes the proof of Theorem ^ 
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